Simulating three-dimensional views using depth relationships among planes of content

ABSTRACT

Approaches enable image content (e.g., still or video content) to be displayed in such a way that the image content will appear, to a viewer, to include portions with different locations in physical space, with the relative positioning of those portions being determined at least in part upon a current relative position and/or orientation of the viewer with respect to the device, as well as changes in that relative position and/or orientation. For example, relationship pairs for image content capable of being displayed on a display screen can be determined. Based on the relationship pairs, a node hierarchy that includes position information for planes of content that include the image content can be determined. The position information can be to render a view of the image content based on the relative position, direction, and/or orientation between the viewer and device to provide a two- or three-dimensional representation of that image content that is appropriate for that viewing angle, giving the impression of a three-dimensional view or display even when the display is in two dimensions.

BACKGROUND

As the capabilities of various computing devices increase, and as peopleare utilizing computing devices for an increasing variety of tasks, theexpectations of users of these devices continues to increaseaccordingly. As an example, an increasing number of applications areattempting to meet these expectations by providing a virtual reality,enhanced reality, or three-dimensional experience. While some devicesutilize three-dimensional displays that require specific hardware, suchas special viewing glasses, these can be expensive and complex, and canprovide varying levels of user satisfaction. A large number of devicesstill utilize conventional two-dimensional displays or provide contentthat is substantially created in two dimensions. While certain shadingor rendering can be utilized to give the impression of three-dimensionalcontent, the content will typically not act like a truethree-dimensional object or scene, as changes in position, orientation,or lighting will generally not be updated realistically in the display.Thus, the virtual nature of the displayed content can be significantlydegraded.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 illustrates an example situation where a user can view contentand interact with a computing device in accordance with variousembodiments;

FIG. 2 illustrates an example state of an interface that can be renderedin accordance with an embodiment;

FIG. 3 illustrates an example state of an interface that can be renderedin accordance with various embodiments;

FIG. 4 illustrates an example system for rendering content on a displayscreen in accordance with various embodiments;

FIGS. 5(a), 5(b), and 5(c) illustrate various states of an interfacethat can be rendered in accordance with various embodiments;

FIGS. 6(a) and 6(b) illustrate various states of an interface that canbe rendered in accordance with various embodiments;

FIGS. 7(a), 7(b), and 7(c) illustrate various states of the mapinformation that can be rendered in accordance with various embodiments;

FIGS. 8(a), 8(b), and 8(c) illustrate various states of levels ofgraphical icons that can move by level in accordance with variousembodiments;

FIGS. 9(a) and 9(b) illustrate various states of book content inaccordance with various embodiments;

FIG. 10 illustrates an example process for updating a display ofinformation to account for orientation changes in accordance withvarious embodiments;

FIG. 11 illustrates an example process for determining a relativeposition of a viewer that can be used in accordance with variousembodiments;

FIG. 12 illustrates an example device that can be used to implementaspects of the various embodiments;

FIG. 13 illustrates example components of a client device such as thatillustrated in FIG. 11;

FIGS. 14(a)-14(f) illustrate example approaches to determining headposition and/or gaze direction that can be used in accordance withvarious embodiments;

FIGS. 15(a) and 15(b) illustrate example approaches to determiningchanges in the relative distance to a user in accordance with variousembodiments

FIGS. 16(a)-16(d) illustrate example approaches to determining changesin the relative viewing angle for a user in accordance with variousembodiments;

FIGS. 17(a) and 17(b) illustrate an example approach to determining therelative position of a user that can be utilized in accordance withvarious embodiments

FIGS. 18(a) and 18(b) illustrate an example approach to determiningdevice motion that can be utilized in accordance with variousembodiments; and

FIG. 19 illustrates an environment in which various embodiments can beimplemented.

DETAILED DESCRIPTION

Systems and methods in accordance with various embodiments of thepresent disclosure may overcome one or more of the aforementioned andother deficiencies experienced in conventional approaches to displayingcontent using an electronic device. In particular, various embodimentsenable image content (e.g., still or video content) to be displayed insuch a way that the image content will appear, to a viewer, to includeportions with different locations in physical space, with the relativepositioning of those portions being determined at least in part upon acurrent relative position and/or orientation of the viewer with respectto the device, as well as changes in that relative position and/ororientation. The content can include various portions, and differentadjustments can be applied to each portion based upon these and/or othersuch changes. These adjustments can include, for example, changes due toparallax or occlusion, which when added to the rendered content inresponse to relative movement between a viewer and a device can enhancethe experience of the viewer and increase realism for content renderedon a two- or three-dimensional display screen.

For example, various embodiments utilize a node hierarchy or other suchhierarchy to enable image content to be displayed to provide a viewerwith an appearance or view of content that appears to be positionedand/or displayed in 3D space. For example, some of the content canappear closer to (or above) a surface of a display screen of a device(and hence the viewer), while other planes of content “fall back” orappear smaller in 3D space, appearing to be further from the surface ofthe display screen. The device can determine relationships, such asposition relationships, between content capable of being displayed onthe display screen. An example relationship can be the relationshipbetween a background interface element, an image element, a headerelement, and text, and can include relationship information such as therelative arrangement between the different content. This can includeposition information, e.g., lateral position information, between thecontent. For example, the background interface element can havepositioned thereon the image element, the header element, and the text.Accordingly, the relationship between these objects would include theposition of the image element (e.g., lateral position), the position ofthe header element, and the position of the text on the backgroundinterface element. Accordingly, using the relationships between thecontent, the content can be organized in a number of different ways. Onesuch way is to organize the content in a hierarch, such as one thatincludes parent and child nodes. Content can be associated with a node,such as a parent node and other content can be associated with childnodes. The nodes can be part of a node hierarchy, as may be provided viathe operating system and/or other software on, or remote from, thecomputing device. Accordingly, using the hierarchy, developers canquickly and efficiently modify the appearance and/or actions of thecontent.

Based on the relationships, the device can determine a node hierarchy.The node hierarchy can include a plurality of nodes, where each node caninclude position information, such as a lateral position relative to adisplay screen and at least a subset of the nodes can include furtherinclude a depth position relative to the display screen. In variousembodiments, the position information can be specified by a developer,where the developer specifies the lateral position and the depthposition. In some embodiments, the device can specify the positioninformation. For example, the device can specify a default depth valuefor the content in the situation where no depth information has beenprovided. The position information can be used by the device to render aview of the content based on the relative position, direction, and/ororientation between the viewer and device to provide a two- orthree-dimensional representation of that content that is appropriate forthat viewing angle, giving the impression of a three-dimensional view ordisplay even when the display is in two dimensions.

Using a camera of the device, at least one image of a viewer of thecomputing device can be acquired, and by analyzing the image, a viewingdirection of the viewer with respect to the computing device can bedetermined. Based on the viewing direction, the device can determine alateral offset for the subset of the plurality of planes of content. Thedevice can then display, on the display screen, the elements based on anassociated one of the plurality of planes of content, where each planeof the plurality of planes of content can be displayed according to arespective lateral position and a respective depth position and thesubset of the plurality of planes of content can be displayed accordingto the respective lateral offset.

In accordance with various embodiments, as the relative position of theviewer and/or orientation of the device changes, the positioninformation for corresponding planes of content is updated, and theupdated position information can be used to adjust the perspective fromwhich the planes of content is rendered to correspond to changes in therelative viewing angle of the viewer. For example, portions of imagecontent (e.g., planes or layers of content) can appear to be positionedand/or displayed in 3D space such that that some of the planes ofcontent appear closer to a surface of the display screen of the device(and hence the viewer), while other planes of content “fall back” orappear smaller in 3D space, appearing to be further from the surface ofthe display screen. As the viewer tilts, rotates, or otherwise changesthe orientation of the device, or as the viewer's relative position ororientation changes with respect to the device, the planes of contentcan appear to translate laterally, move back and forth in apparentdistance from the surface of the screen, or otherwise change shape orappearance. The relative movements can be based upon factors such as thedistance of the viewer to the device, a direction of movement of theuser, a direction of change in orientation of the device, or other suchfactors. The relative movements can be selected such that the differentlayers of content appear to be positioned in three dimensions withrespect to each other, and act appropriately with changes in relativeposition and/or orientation, and thus viewing angle, of the viewer.

In various embodiments, the relative position and/or orientation of aviewer of a computing device can be determined using at least one imagecapture element of the device. For example, the feed from a video cameracan be analyzed to locate a relative position of the viewer in the videofeed, which can be analyzed to determine the relative direction of theviewer. In other embodiments, one or more digital still cameras cancapture images periodically, in response to detected movement of theviewer and/or device, or at other appropriate times, which then can beanalyzed to attempt to determine viewer position, as distance can oftenbe determined in addition to direction when analyzing multiple sourcesof information from different locations. Distance can be determined, forexample, using stereoscopic imaging or proximity sensing, among othersuch options. In some embodiments, infrared (IR) imaging can be used todetect specific features of the viewer, such as the viewer's eyes, foruse in determining and/or tracking the location of the viewer. In stillother embodiments, changes in the orientation and/or position of thedevice can be determined using at least one motion sensor of the device,in order to provide for a higher sampling frequency than might otherwisebe possible using the image information captured by the camera, orotherwise attempt to improve the relative position determinations. Insome situations, a sensor that is remote, separate, or otherwise incommunication with the device can be used to detect a change inorientation and/or position of the device. The orientation informationcan be received at the device from the sensor, and the device can causethe appearance of the interface to be altered based at least in part onthe received orientation and/or position information. Accordingly, aviewer can view and interact with the planes of content, and canmaneuver through the planes of content using various approachesdiscussed herein.

Based at least in part upon the determined direction of the viewer, thedevice can determine a primary viewing angle with respect to the planeof the display screen, and thus the planes of content (e.g., a scene) tobe rendered and displayed on the device. For at least certain types ofcontent, the device can adjust the rendering to provide a two- orthree-dimensional representation of that content that is appropriate forthat viewing angle, giving the impression of a three-dimensional view ordisplay even when the display is in two dimensions. For example, aschanges in the relative position, direction, and/or orientation betweenthe viewer and device are determined, a set of transformation equationsand/or coefficients for those equations to adjust a scale and atranslation for the content can be determined. The transformationequations can be used to adjust the perspective from which the planes ofcontent are rendered to correspond to changes in the relative viewingangle of the viewer. In this way, the equations can be used to determinehow to display or otherwise alter the appearance of the planes ofcontent in 3D space (e.g., such as by bringing an element “forward”toward the front of the display screen or bringing an element “back”from the front of the display screen), laterally, etc. For example, inaccordance with an embodiment, the transformation equations are used toapply a scale effect that mimics that which would accompany aperspective frustum by scaling about a fixed pivot point in the centerof the display screen. As the position of the viewer to the devicechanges, the simulated frustum is skewed to the side by translatingdeeper planes of content laterally based on their depth. For example, alateral motion of a user with respect to a computing device can have anassociated change in angular direction with respect to a normal of adisplay screen of the device. Using basic geometry, the change in anglecan result in different lateral translations of objects in differentplanes of content, based at least in part upon the virtual distancebetween those planes. Planes and objects included in those planes thatare intended to appear closer to the user will then be translated by agreater amount than planes and objects in those planes that are intendedto appear further from the user, in order to provide the impression ofobjects positioned in three-dimensional space.

In at least some embodiments, a computing device can attempt todetermine changes in the relative position, direction, and/ororientation between the viewer and device in order to update theperspective from which the displayed content is rendered or otherwisedisplayed. For example, the device can continue capturing and analyzingimage information to attempt to determine changes in relative positionof the viewer, such as may be based on movement of the viewer and/or thedevice. The device also can utilize information from at least oneorientation or position determining element of the device, such as anaccelerometer or inertial sensor, to assist in detecting motions of thedevice and updating the viewing angle accordingly. These elements alsocan detect changes in orientation of the device, such as throughrotation of the device, even though the relative position between theviewer and the device might not have substantially changed. The displaycan be updated based at least in part upon changes in orientation aswell. By adjusting the perspective from which the image content isrendered to correspond to changes in the relative viewing angle of theuser, a three-dimensional representation can be generated on a two- orthree-dimensional display screen that is consistent, across multipleviewing angles, with actual or virtual three-dimensional content.

The ability to update a perspective of rendered content can provideadditional advantages as well. For example, an object included in theplanes of content might at least partially obscure or occlude anotherobject. Using conventional displays, a viewer would not be able to viewthe occluded content. By enabling the rendering perspective to changebased upon relative position or orientation, a viewer can effectivelylook “around” the occlusion to view the content that was previously notvisible in the display. Further, the amount by which the occlusion movesupon a change in position or orientation can be indicative of a relativeheight or distance between the occlusion and the other content, whichcan be useful for mapping or other such applications.

Various other applications, processes, and uses are presented below withrespect to the various embodiments.

FIG. 1 illustrates an example situation 100 wherein a user 102 isinteracting with a computing device 104. Although a portable computingdevice (e.g., a smart phone, an electronic book reader, or tabletcomputer) is shown, it should be understood that various other types ofelectronic device that are capable of determining and processing inputcan be used in accordance with various embodiments discussed herein.These devices can include, for example, notebook computers, personaldata assistants, video gaming consoles or controllers, portable mediaplayers, and wearable computers (e.g., smart watches, smart glasses,etc.) among others. In this example, the computing device 104 includes acamera 106 positioned on a side or corner of the device such that theimaging element will likely be able to capture image information of atleast a portion of the user while the user is viewing content displayedon the device. For example, the imaging element 106 in FIG. 1 is on thefront of the device such that an angular capture range 108 of theimaging element can image at least a portion of the user while theviewer is viewing content displayed on the display element of theelectronic device. In accordance with various embodiments, being able tocapture image information for the user enables the device to determine arelative position and/or orientation of the user with respect to thedevice and adjust a display of content on the device in accordance withthat position and/or orientation.

For example, the display screen can present the appearance of 3D, or3D-like behavior, but might be a standard 2D display. Accordingly, in atleast some embodiments, a node hierarchy or other such hierarchy can beutilized to group or associate image content (e.g., images, text, etc.)to different planes, levels, or other such groupings of content, wherethe image content included within those planes can be displayed toprovide a viewer with an appearance or view of the content that appearsto be positioned and/or displayed in 3D space. In this way, variousembodiments enable image content (e.g., images, text, advertisements,etc.) included on these planes to appear in 3D, such as by bringing aplane (and the content included on the plane) “forward” or otherwisecausing the plane to appear to be positioned towards the front of thedisplay screen in a 3D display or quasi-three-dimensional rendering on a2D display element of the device, while other planes (and the content onthose planes) appear to be positioned “backwards” or at a greaterdistance from the front of the display screen. Further, the renderingcan utilize 3D mapping information, such as a set of layer depths orz-levels, to determine how to relate various interface planes to eachother.

FIG. 2 illustrates an example 200 of an interface displayed on a displayscreen 202 of a computing device 200. In this example, a user is viewinga conventional 2D representation of a webpage 201. As with manyconventional webpages, the areas of the webpage can be divided intozones or areas depending on the content type, the markup of the webpage,visual classification of the webpage, and/or white space analysis of thewebpage. In this example, portions of image content (e.g., planes orlayers of content, or other interface elements) can include a header204, article text 206, at least one image 208, at least one link 210,advertisements 214, and various other links 216. It should beunderstood, however, that aspects of the various embodiments can be usedwith a variety of types of interface, which can include a wide varietyof different portions of image content.

In a conventional 2D representation, the portions of image content, suchas the interface elements described, can be organized in a number ofdifferent ways. One such way is to organize the interface elements in ahierarchy, such as one that includes parent and child nodes. Aninterface element can be associated with a node, such as a parent nodeand one or more other interface elements can be associated with childnodes. The nodes can be part of a view hierarchy, as may be provided viathe operating system and/or other software on, or remote from, thecomputing device. An example of a view hierarchy can include, forexample, a root node of the view hierarchy, where there may be a layoutcontainer or view group that sets forth how its children are to bearranged for display. Examples of layouts can include frame layouts,linear layouts, relative positioning layouts, grid-based layouts, amongothers. Layout containers can include one or more child layoutcontainers. Child layout containers can include one or more of their ownchild layout containers and one or more of their own view. In someembodiments, a layout can be defined declaratively in a language such asXML and/or programmatically in a language such as Java®. An advantage ofsuch an approach can enable the “look” of an application to be designedwithout particularized knowledge of software programming.

The relationship between the nodes (e.g., parent-child relationship in aview hierarchy) can include position information and the positioninformation can be used to determine the relative position of oneinterface element to another interface element. Accordingly, using thehierarchy, such as any view hierarchy of interface elements, developerscan quickly and efficiently modify the appearance and/or actions of theinterface elements to create an interactive interface. For example,developers can control the view or layout of the interface elements andthe content included in those elements. Developers can further add orremove elements, such as buttons, headers, among other interfaceelements. However, because the positioning information includes relativepositioning information (i.e., the relative position of one interfaceelement to another interface element), and does not include depthinformation (e.g., a position depth or screen-space positioninformation), it becomes difficult to render a layout or view as a 3Dscene with parallax, shadows, and 3D perspective without the absoluteposition of the interface elements relative to the display screen.

Accordingly, in accordance with various embodiments, a node hierarchy orother such hierarchy can be utilized to enable image content to bedisplayed to provide a viewer with an appearance or view of content thatappears to be positioned and/or displayed in 3D space. For example, as aview is built based on a view hierarchy, a listening component or othersuch component can attach “listener” nodes to each view in order toconstruct a parallel node hierarchy that contains screen-space positioninformation. In accordance with various embodiments, a node hierarchycan include, for example, depth information for planes of content. Aview hierarchy can include, for example, the layout information ofcontent. Layout information can include the relative position of content(e.g., where content is positioned relative to other content), the typeof content (e.g., headers, images, text), among other information usedto provide a view of the content. In accordance with variousembodiments, a listener node or component is code that when executed bythe device can cause the device to detect changes in the screen-spaceposition of planes of content due to animations, scrolling, changes inthe relative position, direction, and/or orientation between the viewerand device, and can propagate the updated screen-space coordinates todescendant planes of content. In this way, the parallel node hierarchycan track views that affect the appearance of depth for planes ofcontent. As such, large leaf sub-trees of the hierarchy wherescreen-space positioning does not matter are not included into the nodehierarchy.

Accordingly, image content (e.g., images, text, etc.) can be grouped orotherwise contained or assigned to different planes, levels, or othersuch groupings and the planes of content can enable the image contentincluded within those planes to be displayed to provide a viewer with anappearance or view of the content that appears to be positioned and/ordisplayed in 3D space. In this way, the planes have the effect ofmanipulating the z-position (i.e., the appearance of depth relative to adisplay screen of the computing device) of the plane and the imagecontent included in the plane. The planes of content can be associatedwith a subset of a plurality of nodes of the node hierarchy, where eachnode can include screen-space position data instead of, or addition to,relative position data. Thus, each node can include the screen-spaceposition data for an associated plane, where the screen-space positiondata can include position information for a planes screen-spaceposition. In accordance with various embodiments, a screen-spaceposition can include the position of plane relative to the displayscreen, where the position can include a lateral position (x/ycoordinates) and a depth position (z-position) of the plane relative tothe display screen.

The screen-space coordinates for a respective node can be updated by thelistener nodes based on changes in that nodes corresponding plane'sposition due to animations or scrolling, and/or changes in orientationof the device or user of the user relative to the device. The trackingcan occur automatically. Accordingly, the positioning data provided bythis system can enable several important effects. For example, therelative positions of two views at different depths that do not share adirect parent-child relationship allow shadows to be cast correctlybetween them. It also gives the necessary information to computeperspective effects, as it is necessary to know where each plane is withrespect to the shape of a view frustum. Additionally, if true 3D contentis embedded into the views, the position information allows the 3Dcontent to be oriented correctly based on the 3D content's location in3D space.

For example, FIG. 3 shows an example 320 of an interface 301 displayedon a display screen 302 of a computing device 300. As shown, theinterface includes a number of interface planes or levels of content.Image content can be grouped or otherwise contained or assigned todifferent the interface planes or levels of content. The image contentcan include, for example, a header 304, article text 306, at least oneimage 308, at least one link 310, advertisements 314, and various otherlinks 316. The interface planes can be displayed at different depths (orz-levels) on the display screen 302 of computing device 300. Theseinterface planes may be specified through software, the user, adeveloper of content, etc. In some embodiments, the interface planes maybe provided via the operating system on the device, where differentportions of an image or other content to be displayed can be assigned todifferent planes, levels, etc. The interface planes can enable imagecontent (e.g., images, text, etc.) included within those planes to bedisplayed to provide a viewer with an appearance or view of the contentthat appears to be positioned and/or displayed in 3D space. In someembodiments, an interface plane whose position along the z-axis can bedescribed in relation to either its parent of the display screen.

In this example, image content other links 316 is displayed at thehighest level while advertisements 314 are displayed at the lowestlevel. Accordingly, each plane (and content included on the plane) canbe rendered to correspond to a particular depth, such as may bedescribed in relation to either a planes parent plane or the displayscreen. In some situations, an interface plane can be inside (i.e.,nested) another interface plane. For example, image 308 and link 310 areon a first plane that is inside a second plane that includes articletext 306. In this situation, the second plane that includes the articletext 306 is a parent interface plane to the first plane that includesimage 308 and link 310. As such, the first plane that includes image 308and link 310 can inherent the depth of its parent plane (i.e., thesecond plane) and can be offset by some relative amount from its parentplane. Further, in accordance with various embodiments, by usinginterface planes to specify positioning along the z-axis, all descendentviews may benefit from one or more visual cues to create a sense ofdepth. These effects can include shadow casting, parallax, and depthscaling, among others.

FIG. 4 illustrates an example system for rendering content on a displayscreen in accordance with various embodiments. As shown in FIG. 4,computing device 400 includes a rendering component 402 that can rendera view of an interface on display screen 404. For example, the renderingcomponent can, for example, enable image content (e.g., still or videocontent) to be displayed in such a way that the image content willappear, to a viewer, to include portions with different locations inphysical space, with the relative positioning of those portions beingdetermined at least in part upon a current relative position and/ororientation of the viewer with respect to the device, as well as changesin that relative position and/or orientation. As described, the contentcan include various portions, and different adjustments can be appliedto each portion based upon these and/or other such changes. Theseadjustments can include, for example, changes due to parallax orocclusion, which when added to the rendered content in response torelative movement between a viewer and a device can enhance theexperience of the viewer and increase realism for content rendered on atwo- or three-dimensional display screen. The display screen can, forexample, present the appearance of 3D, or 3D-like behavior, but might bea standard 2D display.

The rendering component can render a view of an interface using layoutand positioning data stored in a 2D layout database 406 and a depthlayout database 408. Although these databases are shown as separate, invarious embodiments, the data included in these databases can be storedin the same storage component. Further, in accordance with variousembodiments, a portion or all of the information stored in thesedatabases can be stored remotely and accessed by the device over atleast one network connection. The 2D layout database can includeportions of image content, such as the interface elements described, andcan be organized in a number of different ways. One such way is toorganize the interface elements in a hierarchy, such as one thatincludes parent and child nodes. An interface element can be associatedwith a node, such as a parent node and one or more other interfaceelements can be associated with child nodes. The nodes can be part of anode hierarchy, as may be provided via the operating system and/or othersoftware on, or remote from, the computing device. The relationshipbetween the nodes (e.g., parent-child relationship in a node hierarchy)can include position information and the position information can beused to determine the relative position of one interface element toanother interface element.

As described, a node hierarchy or other such hierarchy can be utilizedto enable image content to be displayed to provide a viewer with anappearance or view of content that appears to be positioned and/ordisplayed in 3D space. For example, as a view hierarchy is built, alayout engine 410 or other such component can attach “listener” nodes toeach view in order to construct a parallel node hierarchy that containsscreen-space position information. The node hierarchy including screenspace information can be stored in the depth layout database and used bythe rendering component to render the display. In accordance withvarious embodiments, a node hierarchy can include, for example, depthinformation for planes of content. A view hierarchy can include, forexample, the layout information of content. Layout information caninclude the relative position of content (e.g., where content ispositioned relative to other content), the type of content (e.g.,headers, images, text), among other information used to provide a viewof the content. In accordance with various embodiments, a listener nodeor component is code that when executed by the device can cause thedevice to detect changes in the screen-space position of planes ofcontent due to animations, scrolling, changes in the relative position,direction, and/or orientation between the viewer and device, and canpropagate the updated screen-space coordinates to descendant planes ofcontent.

The screen-space coordinates for a respective node can be updated by thelistener nodes based on changes in that nodes corresponding plane'sposition due to animations or scrolling, and/or changes in orientationof the device or user of the user relative to the device. The trackingcan occur automatically. In accordance with various embodiments, apositioning element 412 and an orientation determining element 414 canbe used to determine changes in the relative position of the viewerand/or orientation of the device, and accordingly, the positioninformation for corresponding planes of content can be updated, and theupdated position information can be used to adjust the perspective fromwhich the planes of content is rendered to correspond to changes in therelative viewing angle of the viewer. As described, portions of imagecontent (e.g., planes or layers of content) can appear to be positionedand/or displayed in 3D space such that that some of the planes ofcontent appear closer to a surface of the display screen of the device(and hence the viewer), while other planes of content “fall back” orappear smaller in 3D space, appearing to be further from the surface ofthe display screen. As the viewer tilts, rotates, or otherwise changesthe orientation of the device, or as the viewer's relative position ororientation changes with respect to the device, the planes of contentcan appear to translate laterally, move back and forth in apparentdistance from the surface of the screen, or otherwise change shape orappearance. The relative movements can be based upon factors such as thedistance of the viewer to the device, a direction of movement of theuser, a direction of change in orientation of the device, or other suchfactors. The relative movements can be selected such that the differentlayers of content appear to be positioned in three dimensions withrespect to each other, and act appropriately with changes in relativeposition and/or orientation, and thus viewing angle, of the viewer.

FIG. 5(a) shows an example state 501 of an interface where image contentis grouped or otherwise contained or assigned to different planes,levels, or other such groupings. As descried, these groupings may bespecified through software, the user, etc. and can be arranged atvarious depths. In some embodiments, the groupings may be provided viathe operating system on the device, where different portions of an imageor other content to be displayed can be assigned to different layers,levels, etc. As shown in FIG. 5(a), there are three levels of planesarranged according to three different depths, z₀, z₁, and z₂. On thefirst level (i.e., the lowest depth z₀ or the depth appearing to befurthest away from a user and/or the display screen) is interface plane500. On the second level (i.e., z₁ which includes child planes 2 a and 2b) is interface plane 502 and interface plane 504. In this example,interface plane 502 and interface plane 504 are child interface planesof parent interface plane 500. On the third level (i.e., level 3 whichincludes grandchild planes 3 a, 3 b, and 3 c) are interface planes 510,512, and 514. Interface planes 510 and 512 are child planes to interfaceplane 502 and grandchildren planes to interface plane 500. Interfaceplane 514 is a child plane to interface plane 504 and a grandchild planeto interface plane 500. The interface planes can include content, suchas images, text, etc. For example, interface plane 502 can include imagecontent arranged in a 2D layout container 503, interface plane caninclude image content arranged in a 2D layout container 505, interfaceplane 510 can include image content arranged in a 2D layout container511, interface plane 512 can include image content arranged in a 2Dlayout container 513, and interface plane can include image contentarranged in a 2D layout container 515. The interface planes can enablethe image content included within those planes to be displayed toprovide a viewer with an appearance or view of the content that appearsto be positioned and/or displayed in 3D space. It will be appreciated bythose of ordinary skill in the art that a user interface could havefewer or greater depths and/or fewer or greater UI elements than areillustrated in FIG. 5(a). Thus, the depiction of the user interface 501in FIG. 5(a) should be taken as being illustrative in nature and notlimiting to the scope of the disclosure.

In accordance with various embodiments, the interface planes 500, 502,504, 510, 512, and 514 can be organized in a node hierarchy, such asnode hierarchy 540 of FIG. 5(b). In this example, interface plane 500can be at the root of the hierarchy, interface planes 502 and 504 asbranches of the root, and interface planes 510, 512, and 514 as leavesof the hierarchy, with interface planes 510 and 512 branching frominterface plane 502 and interface plane 514 branching from interfaceplane 504. As described, the interface planes can be associated with atleast a subset of nodes of the node hierarchy. For example, at least aportion of the nodes can correspond to interface planes and other nodescorrespond to the image content included in those planes. The contentincluded in those planes can be organized in 2D layout containers (suchas containers 503, 505, 511, 513, and 515). Examples of 2D layouts caninclude frame layouts, linear layouts, relative positioning layouts,grid-based layouts, among others. Two-dimensional layout containers caninclude one or more child 2D layout containers that can include contentsuch as one or more user interface elements. Two-dimensional childlayout containers can include one or more of their own child layoutcontainers which can also include content such as one or more userinterface elements.

In accordance with various embodiments, a node hierarchy can be utilizedto enable image content to be displayed to provide a viewer with anappearance or view of content that appears to be positioned and/ordisplayed in 3D space. In this example, as a view hierarchy is built, alistening component or other such component can attach “listener” nodesto each view in order to construct a node hierarchy that containsscreen-space position information for the interface planes. Asdescribed, a node hierarchy can include, for example, depth informationfor planes of content and a view hierarchy can include, for example, thelayout information of content. Layout information can include therelative position of content (e.g., where content is positioned relativeto other content), the type of content (e.g., headers, images, text),among other information used to provide a view of the content. Thelistener node or component can determine changes in the screen-spaceposition of planes of content due to animations, scrolling, changes inthe relative position, direction, and/or orientation between the viewerand device and can propagate the updated screen-space coordinates todescendant planes of content. In this way, the parallel node hierarchycan track views that affect the appearance of depth for planes ofcontent. For example, the screen-space position of planes affected byscrolling, animation, or some other movement of the planes of contentand/or content on those planes of content can be determined, as well asthe relative position, direction, and/or orientation between the viewerand device, and the changes in position (e.g., depth) for each effectedplane of content can be updated in the respective nodes. In accordancewith various embodiments, this can include propagating the updatedscreen-space coordinates to descendant planes of content.

As shown in FIG. 5(b), the nodes of the node hierarchy correspond to theplanes of content that are positioned at different levels. Image content(e.g., images, text, etc.) can be grouped or otherwise contained orassigned to different planes, levels, or other such groupings of contentwhile working within a conventional 2D framework by injecting containercontrols with appropriate attributes into layout files. As shown on thenode hierarchy, the circles correspond to planes of content and therectangles correspond to content (e.g., text, images, animations, etc.)included in a respective plane. The planes of content can be associatedwith a subset of a plurality of nodes of the node hierarchy, where eachnode can include screen-space position data. The screen position datacan include, for example, a lateral position (x/y coordinates) and adepth position (z-position) of the plane relative to the display screen.The nodes corresponding to the content can include screen-space positiondata as well, which can be updated due to movement of the content causedby animations and/or movement of the content caused by a user'sinteraction with the interface (e.g., scrolling or adjusting a scale ofthe interface).

In accordance with various embodiments, the screen-space position for arespective node can be updated by listener nodes based on changes inthat nodes corresponding plane's position due to animations orscrolling, and/or changes in orientation of the device or user of theuser relative to the device. As described, a listener node can monitorthe movement of content and planes of content. The screen-space positioncan be updated for nodes due to changes in orientation of the deviceand/or a change in a position of a viewer of the device. Additionally,in various situations, content may animate and/or the position ofcontent may move due to scrolling and/or animations. In this situation,the screen-space position of the plane of content that includes thecontent as well as any planes of content descending from the plane ofcontent, can be updated. In accordance with various embodiments, thepositioning data provided by this system can enable several importanteffects. For example, the relative positions of two views at differentdepths that do not share a direct parent-child relationship allowshadows to be cast correctly between them. It also gives the necessaryinformation to compute perspective effects, as it is necessary to knowwhere each plane is with respect to the shape of a view frustum.Additionally, if true 3D content is embedded into the views, theposition information allows the 3D content to be oriented correctlybased on the 3D content's location in 3D space. Animations can includecontent rotating, moving laterally, moving towards and away the displayscreen, etc.

For example, FIG. 5(c) shows an example state 560 of an interface wherebased on a change in the relative position, direction, and/ororientation between a viewer and the device, the display of theinterface planes (and content included in those planes) is updated. Forexample, the screen-space position of planes affected by scrolling,animation, or some other movement of the planes of content and/orcontent on those planes can be determined, as well as the relativeposition, direction, and/or orientation between the viewer and device,and the changes in position (e.g., depth) for each effected plane ofcontent can updated in the respective nodes. Accordingly, the nodehierarchy can be redrawn or rendered to correspond to the change. Inthis example, interface planes 500, 502, 504, 510, 512, and 514 can eachbe transformed according to a rotation, scale, translation, perspectiveprojection, among other possibilities, based on the change so as to givethe appearance that the interface planes exist in a 3D environment. Asdescribed, interface planes have the effect of manipulating thez-position (i.e., depth position relative to the display screen) of itand its content. In various embodiments, this can be expressed visuallythrough scale and parallax effects. Parallax is the situation where ifthe relative position, direction, and/or orientation between a viewerand the device is changed, the interface planes and their content can beautomatically translated by an appropriate amount to create a parallaxeffect. The translation effect exhibited by a given interface plane isproportional to its distance from the display screen. As the distanceincreases, interface planes can be translated by a greater amount.Interface planes with a zero depth are right up against the displayscreen and therefore may not exhibit parallax effects. In the situationof depth scaling, interface planes and their contents can be scaled toappear larger if they are closer to the camera. Interface planes andtheir content that are deeper into the background can be renderedsmaller and can be offset based on the relative position, direction,and/or orientation between the viewer and the device to create acompelling depth illusion as the orientation changes.

An example of this is shown in FIG. 5(c), where interface plane 514 iscapable of partially obscuring interface plane 502 as well as interfaceplane 500 located a lower level. As further shown, as the relativeposition of the viewer and/or orientation of the device is changed fromthe position in FIG. 5(a) to the position in FIG. 5(c), the positioninformation for corresponding planes of content can be updated, and theupdated position information can used to adjust the perspective fromwhich the planes of content is rendered to correspond to changes in therelative viewing angle of the viewer. For example, as the viewer tilts,rotates, or otherwise changes the orientation of the device, or as theviewer's relative position or orientation changes with respect to thedevice, the planes of content can appear to translate laterally, moveback and forth in apparent distance from the surface of the screen, orotherwise change shape or appearance. The relative movements can bebased upon factors such as the distance of the viewer to the device, adirection of movement of the user, a direction of change in orientationof the device, or other such factors. The relative movements can beselected such that the different planes of content appear to bepositioned in three dimensions with respect to each other, and actappropriately with changes in relative position and/or orientation, andthus viewing angle, of the viewer. In accordance with variousembodiments, content included in an interface plane can inherent depthrelated effects (e.g., translation and scaling effects). For example,content 509 which lies flat against interface plane 504 can inherent anychange in translation or scale do to its relationship (i.e., beingincluded in the interface plane) with interface plane 504. Similarly,content 511 can inherent any change in translation and/or scale do toits relationship with interface plane 510 and content 513 can inherentany change in translation and/or scale do to its relationship withinterface plane 512. As described, content can include images, text,appearance effects such as highlighting, color, shading, etc.

In accordance with various embodiments, the position information can beused to render shadows based on an intersection of light from a virtuallight source with one of the planes of content. In accordance withvarious embodiments, a virtual light source can be positioned within themodel used by the renderer from which light appears to come from whencontent is rendered. The virtual light source can appear to bepositioned in any number of positions where the light from the lightsource can be used to render shadows. In some situations the virtuallight source can have a fixed position or can have a position thatattempts to mimic real light sources. Example positions can includebehind the viewer of the computing device, from one of the corners ofthe computing device, among others. FIG. 6(a) shows an example state 601of an interface that includes three levels of planes. On the first levelis interface plane 600. On the second level is interface plane 606 andinterface plane 608. Interface plane 606 and interface plane 608 arechild interface planes of parent interface plane 600. As shown,interface plane 606 can cast a shadow 607 on interface plane 600, andinterface plane 604 can cast a shadow 605 on interface plane 600. On thethird level is interface planes 610, 612, and 614. Interface planes 610and 612 are child planes to interface plane 606 and cast respectiveshadows 660 and 670 on interface plane 606. Interface plane 614 is achild plane to interface plane 604 and casts shadow 680 on interfaceplane 604.

As the screen-space position of planes affected by scrolling, animation,or some other movement of the planes, as well as a change in therelative position of the viewer and/or orientation of the device, theposition information for corresponding planes of content is updated, andthe updated position information is used to adjust the perspective fromwhich the planes of content is rendered to correspond to changes in therelative viewing angle of the viewer. Further, shadows and shading canbe adjusted to be appropriate for the perspective. For example, as theviewer tilts, rotates, or otherwise changes the orientation of thedevice, or as the viewer's relative position or orientation changes withrespect to the device, the planes of content can appear to translatelaterally, move back and forth in apparent distance from the surface ofthe screen, or otherwise change shape or appearance. The relativemovements can be based upon factors such as the distance of the viewerto the device, a direction of movement of the user, a direction ofchange in orientation of the device, or other such factors. The relativemovements can be selected such that the different planes of contentappear to be positioned in three dimensions with respect to each other,and act appropriately with changes in relative position and/ororientation, and thus viewing angle, of the viewer. Additionally, theplanes of content might be lighted or shaded according to one or morelight sources near the device, and the lighting direction can be updatedin response to movement of the device to shade the objects in the imageaccording to the position of the light source.

For example, FIG. 6(b) shows an example state 660 of an interface, usingthe known relative location of a virtual light source and the change inorientation and/or the screen-space position of planes due to scrolling,animation, or some other movement of the planes, the device candetermine the proper direction to the light source in the neworientation, and as such can generate shadows for the interface planeselements based at least in part upon the lighting direction, as would beconsistent for the current user viewing angle. The ability to adjustshadowing with the changes in display in a realistic way can help toenhance the user experience, and can also help the user to betterunderstand the direction and/or texture of, for example, atwo-dimensional representation of a three-dimensional element. As shown,the perspective from which the planes of content are rendered isupdated, as well as the perspective of shadows cast from those planes.In this example, as the perspective of interface elements 606, 610, 612,and 614 is updated, their respective shadows 607, 660, 670, and 680 areupdated.

In a conventional layout, the draw or view of the layout of theinterface is in a fixed order. However, in the situation where viewscontain interface planes with content at different depths, the desireddraw order becomes an important factor to ensure correct timing torender shadows. In accordance with various embodiments, draw order isthe order in which images and other objects are displayed on a displayscreen of a computing device. In some situations, a default order or anorder set by a developer of the interface. In this example, the draworder of interface planes can begin with the interface plane at thelowest level. Accordingly, in this example, plane 600 is drawn firstbecause it is at level 1. In accordance with various embodiments, whenan interface plane's draw code path is executed, that plane sets astatic flag indicating that it is currently drawing. Next, shadowscasted on that interface plane are drawn before drawing any planes atthe second level. In this example, shadows 607 and 605 are drawn beforeinterface planes 604 and 606 are drawn. Any descendant that is at thesame depth is drawn. The descendant planes can be drawn in a defaultorder or an order specified by a developer. In this example, descendantor child interface planes 604 and 606 are drawn. If while drawinginterface planes 604 and 606, any of planes 610, 612, 614 begin to draw,these interface planes will first check the state of its ancestorinterface planes (e.g., interface planes 604 and 606). If an ancestorplane is drawing, the interface plane attempting to draw (e.g., one ofinterface planes 610, 612, or 614) will stop drawing and add itselfalong with any state needed to a priority queue based on depth to deferits rendering until a later time. Once all descendants have either drawnor deferred, a current interface plane may render any shadows being caston it and then may set its flag to indicate it is ready to draw a newlevel and begin to iterate through the deferred interface planes indepth order. For example, upon drawing interface planes 604 and 606,shadows 660, 670 and 680 are drawn on the respective interface plane.Finally, interface planes 610, 612, and 614 are drawn to complete theprocess.

FIG. 7(a) illustrates an example device 700 displaying map content on adisplay element 702 of the device. In this example, the user has enteredan address into a mapping application, and mapping information isgenerated for display on the display element 702, including a pin ormarker 710 indicating the approximate location of the address on the mapregion and pin or marker 711 indicating the user's home. In thisexample, the pin or markers 710 and 711 are rendered at a depth abovethe map. However, when the user (not shown) is in a default position (orwithin a default range of positions) with respect to the device, such assubstantially in front of the display screen, the pins or markers appearrelatively flat. It should be noted that approaches to locating anaddress or location and generating map information are well known in theart and, as such, will not be discussed herein in detail.

As discussed, it can be desirable in at least certain embodiments toenhance the realism of such a situation as much as possible. One way isto add shading to the image such that the pin and buildings displayedappear to be three-dimensional objects. Generally, the shadows arerendered from a fixed direction and applied to a particular view, suchas a top-down view as illustrated. If the user moves the device, ormoves relative to the device, however, the shading will not change andthe perspective of the device will not adjust to show buildings or thepin from appropriate views resulting from the rotation, which can takethe viewer out of the experience. Similarly, the appearance of the itemswill not adjust if the user moves relative to the device, such that theuser will be aware that the display is a two-dimensional rendering.

Systems and methods in accordance with various embodiments can takeadvantage of any of a number of elements that can be used to determinechanges in relative position and/or orientation between a user and anelectronic device. For example, the device 700 in FIG. 7(a) includes animaging element 704 which can be used to capture image information fordetermining a relative position or direction of a user as mentionedabove. An orientation-determining element 706, such as an accelerometer,electronic gyroscope, or inertial sensor, can determine changes in theposition or orientation of the device. Other input elements 708, such asmicrophones or proximity sensors, can be used as well in otherembodiments. The information from at least some of these elements can beanalyzed to determine a current viewing angle from the perspective ofthe user. By determining the current viewing angle, for example, thedevice can render content that corresponds substantially to athree-dimensional view of the content from the perspective of the user.

For example, FIG. 7(b) illustrates an example orientation of the device700 upon the device being rotated along a primary axis of the device,although a similar change in relative orientation could result frommotion of the user as will be discussed in more detail later herein. Ascan be seen, the rotation of the device triggered a corresponding changein the map information 704 displayed on the device. For example, insteadof seeing a top view of each building, the sides of various buildingsare displayed corresponding to the current viewing angle of the user.Thus, even though the display is a two-dimensional display, the renderedview of a building can be such that the display will present a view ofthat building that is similar to what the user would see if viewing athree-dimensional version of a building from the current viewing angle.In this example, where the left edge of the device is rotated out of theplane of the Figure, the left side of various buildings is rendered(along with a portion of the roof or other perspective-appropriateportions) based on the viewer direction being substantially orthogonalto the plane of the Figure.

In FIG. 7(b), the rendering of the location pins 702 and 711 have alsoupdated accordingly. In FIG. 7(a), the pins were shown in asubstantially top-down view. In FIG. 7(b), the location pins 702 and 711are rendered to appear to be above the map content and rendered for thecurrent viewing angle of the user. As described, portions of imagecontent, such as the location pins 702 and 711, can be organized in ahierarchy, such as one that includes parent and child nodes. Thelocation pins can be associated with a node, where the nodes can be partof a node hierarchy. The relationship between nodes (e.g., parent-childrelationship in a node hierarchy) can include position information andthe position information can be used to determine the relative positionof one interface element to another interface element. The positioninformation can also include screen-space position data that can includeposition information for a pins screen-space position, such as a lateralposition (x/y coordinates) and a depth position (z coordinate) of theinterface element relative to the display screen. In this example, themap content can be at first level (i.e., be associated with a firstdepth) and the pins can be at second level, where the second level is“taller” than the first level and content associated with the secondlevel appears closer to the display screen. In addition to changing theway the pin is displayed, the user can now view information that mighthave previously been hidden or occluded by the pin in the top view. Forexample, in FIG. 7(a) the W in “Washington Street” was occluded by thelocation of the pin. In FIG. 7(b), the rotation of the device hasresulted in the rendering of the pin changing to reflect the currentviewing angle, which also results in the W in Washington Street nowbeing viewable by the user. Thus, the rendering changes not only theperspective of various elements but can also move those elementsappropriately relative to any other elements or occluded portions inorder to further provide the sense of a three-dimensional world. A userthus can adjust the relative orientation between the user and the deviceto view information occluded by an object, which may not have beenpossible in conventional approaches without manually moving or removingthe pin.

Since the user would not have been able to see the full name ofWashington Street in either of the previous orientations, the user cancontinue to adjust the relative orientation of the device until thedesired information is viewable. For example, in situation 750 of FIG.7(c) the user can tilt the top of the screen forward out of the plane ofthe Figure, causing a different rendering of objects in the image. Inthis example, a side of the buildings towards the top of the device canbe seen, and a “height” of the pin 702 is adjusted based upon the neworientation. In this example, the name of the street that was previouslyoccluded now can be seen in the displayed image information, as“Washington Street” is now visible in the image. Further, the positioninformation can be used to render shadows based on an intersection oflight from a virtual light source with one of the planes of content.

Another advantage of changing the orientation is that the user can alsoview different angles of occlusions that might not otherwise have beenobvious to the user. For example, in FIGS. 7(a) and 7(b) there is notmuch distinction in the display of Washington Street and Lincoln Avenue,other than their respective locations. In the rendering of FIG. 7(c),however, the orientation illustrates that Lincoln Avenue is in fact araised street 752, as the position of the street shifts upon orientationchange due to the street being at a different distance or plane.Further, shadowing 754 or other elements (e.g., posts or arches) can beadded to further illustrate the difference in location and perspective.Using conventional top-down views, the user might not have been able todiscern that Lincoln Street was actually above the other nearby streets,and could not be turned directly onto from either of the one way streetsthat cross under Lincoln Avenue.

An approach in accordance with various embodiments can instead utilizelayers of graphical elements that can move at different rates withrespect to each other, providing a sense of depth and three-dimensionalmovement. This can include rendering a view to have at least two (and inmany situations more) different “levels” or z-depths, where the upperlevel of some interface elements is rendered to appear near the outersurface of the display screen and the upper level of other interfaceelements can be rendered to appear at a lower level to the interface(e.g., separated a distance from the outer surface of the displayscreen).

For example, FIG. 8(a) illustrates a display 800 including three layersof graphical elements 802, 804, 806, such as graphical iconsrepresenting applications, folders, shortcuts, or any other type ofobject known or used on electronic devices to access various types offunctionality or information. In this example, a first layer of elements802 is rendered “over” a second layer of elements 804, which is renderedover a third layer of elements 806. It should be understood that therecan be any number of levels including any appropriate number ofelements, and that the ordering of the various layers can be adjusted orselected using any of a number of potential approaches, such as sortingor navigating the various layers. In this example, the elements of eachlayer are rendered with a different relative size, providing to the usera sense of distance of the various elements. In various embodiments, thedevice can use information such as the field of view of the camera, aswell as the position of the user's head or eyes to determine a currentpoint of view of a user, and the point of view can be used to render aninterface on a display screen or other such element of the computingdevice. The rendering can update as the determined point of view changesas a result of movement of the user and/or the computing device. Therendering can utilize 3D mapping information, such as a set of layerdepths or z-levels, to determine how to relate various interfaceelements to each other.

In order to enhance that sense of distance, as well as to provide asense of space and enable the user to obtain different views of thevarious elements, the layers can also move laterally with respect toeach other at different rates, where the rate of movement is coupledwith the relative size for that layer. For example, in FIG. 8(b) therehas been relative movement between the user and the device on which theimage information is displayed. As can be seen, the first layer ofelements 802 that is “closest” to the user has moved by the greatestamount. The second layer of elements 804 has moved by a smaller amount,representative of their respective distances, with the third layer ofelements 806 moving the least, if at all. In fact, in embodiments whereit is desired to keep the information substantially centered on thedisplay, the third layer of elements might actually move in the oppositedirection, as illustrated in FIG. 8(b), although the net relativemovement between could remain the same in either approach. As can beseen, in addition to providing a sense of three-dimensional space, theability to rotate the view enables different elements to be seen, whichcan help the user to locate and/or navigate to an element of interest,which might otherwise be hidden or occluded by an overlying element. Formap views, for example, each block of buildings might be assigned to adifferent layer, enabling the user to make distance determinations andmore accurately determine location information from thequasi-three-dimensional view.

In various embodiments, the device can adjust the appearance of shadowsassociated with the relevant interface elements to make those elementsappear to be higher in the interface, as well as to give a 3D appearanceas each shadow can move in position relative to an associated element asthe point of view changes, for example, to give the impression of anactual shadow being case by the relevant element. Further, the interfacecan render sidewalls or other elements that appear to provide a depth ofthe interface element from the point of view of the user, and the extentand shape of these elements can adjust with changes in point of view, aswell as an orientation of the device. Various other behaviors can beused as well to mimic 3D behavior as well as an appearance of stackedinterface elements. In at least some interfaces, there might be morethan three levels, and the amount of shadowing, color adjusting, andother such aspects can depend at least in part upon the level with whichthe element is associated.

In some embodiments, the graphical elements, as shown in display 820 ofFIG. 8(b) can be three-dimensional blocks that can be rotatedindividually as well, in order to show a least a side portion asdiscussed elsewhere herein. In this example, however, the elements areessentially “flat” or otherwise unable to rotate individually, with thethree-dimensional feel being generated primarily by the differences inlateral translation. While the inability of the individual elements torotate can potentially lessen the three-dimensional experience for someusers with respect to rotatable elements, the amount of processingcapacity can be significantly less and the rendering time less for atleast some devices, which can enhance the overall user experience. Asfurther shown in display 840 of FIG. 8(c), as the relative position,direction, and/or orientation between the viewer and device is changed,the rendering of the displayed content is updated. For example, in FIG.8(c), the view has been updated due to a change in the orientation ofthe device to show a side view of the elements 802, 804, and 806.

FIGS. 9(a) and 9(b) illustrate different views of an electronic book(e-book) 900, or similar content, that can be displayed on an electronicdevice using various approaches discussed herein. In FIG. 9(a), the user(not shown) is in a default position (or within a default range ofpositions) with respect to the device, such as substantially in front ofthe display screen. Accordingly, the user can obtain a conventional viewof the text 902 in the e-book at that viewing angle. In this view, thelinked text 920 and 922 (i.e., text linking to other content) appearsflat and otherwise part of the other text. While reading the e-book, theuser might want to obtain certain information, such as how far the useris from the end of the book, how far it is until the next chapter, orexpose or otherwise emphasize text that links to other content and/or ishighlighted (e.g., underlined or bolded). Accordingly, a user can changea viewing angle of the user with respect to the device, such as byrotating the device or moving the user's head, to view a “side” of theelectronic book on the display. As illustrated in FIG. 9(b), the usercan tilt the device to see a view that includes a representation of theedges 904 of the pages of the book between the current page and the endof the book. Additionally, titling the device causes the links to appearto be above the text on the page, and as such, the user can quicklyidentify the links. Such an approach can provide value to the user, andalso potentially make the e-book experience more like reading an actualbook. Additional information can be added as well. For example, the edgeof the book can include not only an indicator 908 of the end (i.e., backcover) of the book, but can also include other indicators 906 to certainsections as well. For example, the side of the e-book can includeindicators marking the location of the next chapter and/or subsequentchapters, as well as the location of various notes, bookmarks,highlights, etc. Using such an approach, a user can tilt the device tosee how far until the end of the chapter, for example, to determinewhether to continue reading until the end of the chapter or end thecurrent viewing session at a different location. In some embodiments,the rendered image can also be manipulated (e.g., stretched or otherwisedeformed) in order to make the view of an object from the perspective ofthe user seem as if the display screen is a piece of glass through whichthe user is looking, rather than a conventional display screen in whichthings become increasingly compressed as the viewing angle increases.

FIG. 10 illustrates an example of a first portion 1000 of a process forproviding a relative orientation-based image display that can be used inaccordance with various embodiments. It should be understood that, forany process discussed herein, there can be additional, fewer, oralternative steps performed in similar or alternative orders, or inparallel, within the scope of the various embodiments unless otherwisestated. In this example, position tracking of a viewer is activated 1002on the device. In some embodiments a user must activate this modemanually, while in other modes the device can activate the modeautomatically when a person is detected nearby. Other modes ofactivation are possible as well, such as upon a user opening a specificapplication on the device. When the position tracking is active, thedevice can begin imaging 1004 around the device, whether in alldirections, some directions, a specific range of directions, or adirection substantially toward a determined viewer. As discussedelsewhere herein, in some embodiments the imaging will involve ambientlight image or video capture, while in other embodiments a device canutilize infrared imaging, heat signature detection, or any other suchapproach. The device can analyze 1006 the captured image information toattempt to locate features of a user, or at least a person nearby, wherethose features in some embodiments include at least the eyes, nose, orhead of a user. In some embodiments, the device will attempt to locatean object that is shaped like a human head and that contains twoeye-like features. In other embodiments, facial recognition or any othersuch algorithm can be used to attempt to determine the presence of ahuman head, or other portion or feature of a user, in the field of viewof at least one of the imaging elements.

Once the user features are located, the device can attempt to determine1008 aspects or information relating to those features. In this example,the determined aspects can be used to attempt to determine 1010 arelative orientation between the device and the user, as well as theorientation of those features relative to the device in at least someembodiments, which can be useful in determining information such as thepresent viewing location of a user. Image content can be displayed 1012based on the determined viewpoint. As described, image content (e.g.,still or video content) can be displayed in such a way that the imagecontent will appear, to a viewer, to include portions with differentlocations in physical space, with the relative positioning of thoseportions being determined at least in part upon a current relativeposition and/or orientation of the viewer with respect to the device, aswell as changes in that relative position and/or orientation. Thecontent can include various portions, and different adjustments can beapplied to each portion based upon these and/or other such changes. Forexample, in accordance with various embodiments, portions of imagecontent (e.g., planes or layers of content) can appear to be positionedand/or displayed in 3D space such that that some of the planes ofcontent appear closer to a surface of the display screen of the device(and hence the viewer), while other planes of content “fall back” orappear smaller in 3D space, appearing to be further from the surface ofthe display screen. The determined aspects then can be monitored 1014over time, such as by continuing to capture and analyze imageinformation to determine the relative position of the user and/ororientation of the device. In at least some embodiments, anorientation-determining element such as an accelerometer or electronicgyroscope can be used to assist in tracking the relative location of theuser and/or current relative orientation of the device. A change in theaspect, such as a change in position or orientation, can be determined1016, and the device can determine 1018 whether that change requires anadjustment to the image to be displayed. For example, an applicationmight require the device to be rotated a minimum amount before adjustingthe displayed image content, such as to account for a normal amount ofuser jitter or other such movement that may not be intended as input.Similarly, certain embodiments might not utilize continuous rotation,but might change views upon certain degrees of change in relativeorientation. If the orientation change is sufficient to warrant anadjustment, the device can determine and perform 1020 the appropriateadjustment to the content that is displayed in the image information,such as to adjust the screen-space position of interface elementsincluded in the image information.

For example, as described, the image can include portions of imagecontent (e.g., planes or layers of content). Each plane of content canbe an interface element, such as a shape, text, object, etc. Asdescribed, the interface elements can be organized in a hierarchy ofnodes that can include at least one parent node and one or more childnodes. The relationship between the nodes (e.g., parent-childrelationship in a node hierarchy) can include position information andthe position information can be used to determine the relative positionof one interface element to another interface element. Accordingly,using the hierarchy, such as any node hierarchy of interface elements,developers can quickly and efficiently adjust the appearance and/oractions of the interface elements to create an interactive interface.Further, each node can include screen-space position data. As described,screen-space position data can include the position of an interfaceelement relative to the display screen, where the position can include alateral position (x/y coordinates) and a depth position (z coordinate)of the interface element relative to the display screen. Thescreen-space coordinates for the nodes can be updated based on changesin a nodes position due to animations or scrolling, and/or changes inorientation of the device or user of the user relative to the device,and the updated position information can be used to adjust a view of theimage information. Further, the position information can be used torender shadows based on an intersection of light from a virtual lightsource with one of the planes of content.

As an example of one such adjustment, FIG. 11 illustrates a secondportion 1100 of a process for modifying the image in response to adetermined change in orientation that can be used in accordance withvarious embodiments. During operation, an electronic device can acquire1102, using a camera of the device, at least one first image of a viewerof the device. The device can determine 1104 (and monitor over time), byanalyzing the first image, a first viewing direction or viewing angle ofthe viewer with respect to the device. The device can determine 1106relationships between content capable of being displayed on the displayscreen. An example relationship can be the relationship between abackground interface element, an image element, a header element, andtext, and can include relationship information such as the relativearrangement between the different content. This can include positioninformation, e.g., lateral position information, between the content.

Based on the relationships, the device can generate 1108 a nodehierarchy. The node hierarchy can include a plurality of nodes, whereeach node can include position information, such as a lateral positionrelative to a display screen and at least a subset of the nodes caninclude further include a depth position relative to the display screen.In various embodiments, the position information can be specified by adeveloper, where the developer specifies the lateral position and thedepth position. In some embodiments, the device can specify the positioninformation. For example, the device can specify a default depth valuefor the content in the situation where no depth information has beenprovided. The position information can be used by the device to render aview of the content based on the relative position, direction, and/ororientation between the viewer and device to provide a two- orthree-dimensional representation of that content that is appropriate forthat viewing angle, giving the impression of a three-dimensional view ordisplay even when the display is in two dimensions.

Position information for a plurality of planes of content can bedetermined 1110, where the position information for each plane of theplurality of planes of content can include a first lateral position anda first depth position relative to the display screen. As described,planes or layers of content, or other interface elements can include aheader, article text, at least one image, at least one link,advertisements, and various other links. It should be understood,however, that aspects of the various embodiments can be used with avariety of types of interface, which can include a wide variety ofdifferent portions of image content.

An application executing on the device (or remote to the device) canutilize mapping, the position information, or other such data to renderimage content from a perspective associated with the first viewingdirection. For example, the device can display 1112, on the displayscreen, the plurality of planes of content, where each plane of theplurality of planes of content is displayed according to the respectivefirst lateral position and the respective first depth position, and thesubset of the plurality of planes of content further being displayedaccording to the respective first lateral offset.

Once the viewing direction of the user is determined, the device canattempt to monitor 1112 or detect changes in the position of contentincluded in the planes of content as well as changes in the planes ofcontent, as may result from scrolling, animation, or some other movementof the planes of content and/or content on those planes of content.Changes in the relative position can be analyzed to determine whetherthe change is actionable 1114, such as where the change meets a minimummovement threshold. In some embodiments, small movements might notresult in adjustments in the display, in order to account for jitter orsubtle variations due to the user holding a device, for example, thatare not intended as input to change the perspective. In variousembodiments, there also must be a minimum amount of movement in order tojustify the re-rendering of the displayed image content.

If there is no actionable movement, the device can continue to monitorthe relative position of the user. If there is actionable movementdetected, the device can attempt to determine the new relative positionof the content included in the planes of content as well as changes inthe position of the planes of content using any of the approachesdiscussed or suggested herein. The changes in position (e.g., depth) foreach effected plane of content can be updated in the respective nodes.In accordance with various embodiments, this can include propagating theupdated screen-space coordinates to descendant planes of content. Forexample, as described, as a view hierarchy is built, a layout engine orother such component can attach “listener” nodes to each view in orderto construct a parallel node hierarchy that contains screen-spaceposition information. In accordance with various embodiments, thelistener nodes can cause the device to detect changes in thescreen-space position of planes of content due to animations, scrolling,changes in the relative position, direction, and/or orientation betweenthe viewer and device, and can propagate the updated screen-spacecoordinates to descendant planes of content. The screen-spacecoordinates for a respective node can be updated by the listener nodesbased on changes in that nodes corresponding plane's position due toanimations or scrolling, and/or changes in orientation of the device oruser of the user relative to the device. The tracking can occurautomatically. The device can then update the display of at least asubset of the plurality of planes of content, which can include changingthe appearance of content (e.g., rotated, stretched, compressed,translated, etc.) to provide a consistent quasi-three-dimensional viewas discussed elsewhere herein. As discussed, additional information canbe added as well, such as shadowing from a nearby light source. In atleast some embodiments, an application can attempt to provideconsistency in the rendering and shading from any of a number ofdifferent viewing angles consistent with a three-dimensional display,even when the element used to display the image information istwo-dimensional in nature.

FIG. 12 illustrates front and back views of an example electroniccomputing device 1200 that can be used in accordance with variousembodiments. Although a portable computing device (e.g., a smartphone,an electronic book reader, or tablet computer) is shown, it should beunderstood that any device capable of receiving and processing input canbe used in accordance with various embodiments discussed herein. Thedevices can include, for example, desktop computers, notebook computers,electronic book readers, personal data assistants, cellular phones,video gaming consoles or controllers, television set top boxes, andportable media players, among others.

In this example, the computing device 1200 has a display screen 1202(e.g., an LCD element) operable to display information or image contentto one or more users or viewers of the device. The display screen ofsome embodiments displays information to the viewers facing the displayscreen (e.g., on the same side of the computing device as the displayscreen). The computing device in this example can include one or moreimaging elements, in this example including two image capture elements1204 on the front of the device and at least one image capture element1210 on the back of the device. It should be understood, however, thatimage capture elements could also, or alternatively, be placed on thesides or corners of the device, and that there can be any appropriatenumber of capture elements of similar or different types. Each imagecapture element 1204 and 1210 may be, for example, a camera, acharge-coupled device (CCD), a motion detection sensor or an infraredsensor, or other image capturing technology.

As discussed, the device can use the images (e.g., still or video)captured from the imaging elements 1204 and 1210 to generate athree-dimensional simulation of the surrounding environment (e.g., avirtual reality of the surrounding environment for display on thedisplay element of the device). Further, the device can utilize outputsfrom at least one of the image capture elements 1204 and 1210 to assistin determining the location and/or orientation of a user and inrecognizing nearby persons, objects, or locations. For example, if theuser is holding the device, the captured image information can beanalyzed (e.g., using mapping information about a particular area) todetermine the approximate location and/or orientation of the user. Thecaptured image information may also be analyzed to recognize nearbypersons, objects, or locations (e.g., by matching parameters or elementsfrom the mapping information).

The computing device can also include at least one microphone or otheraudio capture elements capable of capturing audio data, such as wordsspoken by a user of the device, music being hummed by a person near thedevice, or audio being generated by a nearby speaker or other suchcomponent, although audio elements are not required in at least somedevices. In this example there are three microphones, one microphone1208 on the front side, one microphone 1212 on the back, and onemicrophone 1206 on or near a top or side of the device. In some devicesthere may be only one microphone, while in other devices there might beat least one microphone on each side and/or corner of the device, or inother appropriate locations.

The device 1200 in this example also includes one or more orientation-or position-determining elements 1218 operable to provide informationsuch as a position, direction, motion, or orientation of the device.These elements can include, for example, accelerometers, inertialsensors, electronic gyroscopes, and electronic compasses.

The example device also includes at least one communication mechanism1214, such as may include at least one wired or wireless componentoperable to communicate with one or more electronic devices. The devicealso includes a power system 1216, such as may include a batteryoperable to be recharged through conventional plug-in approaches, orthrough other approaches such as capacitive charging through proximitywith a power mat or other such device. Various other elements and/orcombinations are possible as well within the scope of variousembodiments.

FIG. 13 illustrates a set of basic components of an electronic computingdevice 1300 such as the device 1200 described with respect to FIG. 12.In this example, the device includes at least one processing unit 1302for executing instructions that can be stored in a memory device orelement 1304. As would be apparent to one of ordinary skill in the art,the device can include many types of memory, data storage, orcomputer-readable media, such as a first data storage for programinstructions for execution by the processing unit(s) 1302, the same orseparate storage can be used for images or data, a removable memory canbe available for sharing information with other devices, and any numberof communication approaches can be available for sharing with otherdevices.

The device typically will include some type of display element 1306,such as a touch screen, electronic ink (e-ink), organic light emittingdiode (OLED) or liquid crystal display (LCD), although devices such asportable media players might convey information via other means, such asthrough audio speakers.

As discussed, the device in many embodiments will include at least oneimaging element 1308, such as one or more cameras that are able tocapture images of the surrounding environment and that are able to imagea user, people, or objects in the vicinity of the device. The imagecapture element can include any appropriate technology, such as a CCDimage capture element having a sufficient resolution, focal range, andviewable area to capture an image of the user when the user is operatingthe device. Methods for capturing images using a camera element with acomputing device are well known in the art and will not be discussedherein in detail. It should be understood that image capture can beperformed using a single image, multiple images, periodic imaging,continuous image capturing, image streaming, etc. Further, a device caninclude the ability to start and/or stop image capture, such as whenreceiving a command from a user, application, or other device.

The example computing device 1300 also includes at least one orientationdetermining element 1310 able to determine and/or detect orientationand/or movement of the device. Such an element can include, for example,an accelerometer or gyroscope operable to detect movement (e.g.,rotational movement, angular displacement, tilt, position, orientation,motion along a non-linear path, etc.) of the device 1300. An orientationdetermining element can also include an electronic or digital compass,which can indicate a direction (e.g., north or south) in which thedevice is determined to be pointing (e.g., with respect to a primaryaxis or other such aspect).

As discussed, the device in many embodiments will include at least apositioning element 1312 for determining a location of the device (orthe user of the device). A positioning element can include or comprise aGPS or similar location-determining elements operable to determinerelative coordinates for a position of the device. As mentioned above,positioning elements may include wireless access points, base stations,etc., that may either broadcast location information or enabletriangulation of signals to determine the location of the device. Otherpositioning elements may include QR codes, barcodes, RFID tags, NFCtags, etc., that enable the device to detect and receive locationinformation or identifiers that enable the device to obtain the locationinformation (e.g., by mapping the identifiers to a correspondinglocation). Various embodiments can include one or more such elements inany appropriate combination.

As mentioned above, some embodiments use the element(s) to track thelocation of a device. Upon determining an initial position of a device(e.g., using GPS), the device of some embodiments may keep track of thelocation of the device by using the element(s), or in some instances, byusing the orientation determining element(s) as mentioned above, or acombination thereof. As should be understood, the algorithms ormechanisms used for determining a position and/or orientation can dependat least in part upon the selection of elements available to the device.

The example device also includes one or more wireless components 1314operable to communicate with one or more electronic devices within acommunication range of the particular wireless channel. The wirelesschannel can be any appropriate channel used to enable devices tocommunicate wirelessly, such as Bluetooth, cellular, NFC, or Wi-Fichannels. It should be understood that the device can have one or moreconventional wired communications connections as known in the art.

The device also includes a power system 1316, such as may include abattery operable to be recharged through conventional plug-inapproaches, or through other approaches such as capacitive chargingthrough proximity with a power mat or other such device. Various otherelements and/or combinations are possible as well within the scope ofvarious embodiments.

In some embodiments the device can include at least one additional inputdevice 1318 able to receive conventional input from a user. Thisconventional input can include, for example, a push button, touch pad,touch screen, wheel, joystick, keyboard, mouse, keypad, or any othersuch device or element whereby a user can input a command to the device.These I/O devices could even be connected by a wireless infrared orBluetooth or other link as well in some embodiments. Some devices alsocan include a microphone or other audio capture element that acceptsvoice or other audio commands. For example, a device might not includeany buttons at all, but might be controlled only through a combinationof visual and audio commands, such that a user can control the devicewithout having to be in contact with the device.

In some embodiments, a device can include the ability to activate and/ordeactivate detection and/or command modes, such as when receiving acommand from a user or an application, or retrying to determine an audioinput or video input, etc. In some embodiments, a device can include aninfrared detector or motion sensor, for example, which can be used toactivate one or more detection modes. For example, a device might notattempt to detect or communicate with devices when there is not a userin the room. If an infrared detector (i.e., a detector with one-pixelresolution that detects changes in state) detects a user entering theroom, for example, the device can activate a detection or control modesuch that the device can be ready when needed by the user, but conservepower and resources when a user is not nearby.

A computing device, in accordance with various embodiments, may includea light-detecting element that is able to determine whether the deviceis exposed to ambient light or is in relative or complete darkness. Suchan element can be beneficial in a number of ways. In certainconventional devices, a light-detecting element is used to determinewhen a user is holding a cell phone up to the user's face (causing thelight-detecting element to be substantially shielded from the ambientlight), which can trigger an action such as the display element of thephone to temporarily shut off (since the user cannot see the displayelement while holding the device to the user's ear). The light-detectingelement could be used in conjunction with information from otherelements to adjust the functionality of the device. For example, if thedevice is unable to detect a user's view location and a user is notholding the device but the device is exposed to ambient light, thedevice might determine that it has likely been set down by the user andmight turn off the display element and disable certain functionality. Ifthe device is unable to detect a user's view location, a user is notholding the device and the device is further not exposed to ambientlight, the device might determine that the device has been placed in abag or other compartment that is likely inaccessible to the user andthus might turn off or disable additional features that might otherwisehave been available. In some embodiments, a user must either be lookingat the device, holding the device or have the device out in the light inorder to activate certain functionality of the device. In otherembodiments, the device may include a display element that can operatein different modes, such as reflective (for bright situations) andemissive (for dark situations). Based on the detected light, the devicemay change modes.

Using the microphone, the device can disable other features for reasonssubstantially unrelated to power savings. For example, the device canuse voice recognition to determine people near the device, such aschildren, and can disable or enable features, such as Internet access orparental controls, based thereon. Further, the device can analyzerecorded noise to attempt to determine an environment, such as whetherthe device is in a car or on a plane, and that determination can help todecide which features to enable/disable or which actions are taken basedupon other inputs. If voice recognition is used, words can be used asinput, either directly spoken to the device or indirectly as picked upthrough conversation. For example, if the device determines that it isin a car, facing the user and detects a word such as “hungry” or “eat,”then the device might turn on the display element and displayinformation for nearby restaurants, etc. A user can have the option ofturning off voice recording and conversation monitoring for privacy andother such purposes.

In some of the above examples, the actions taken by the device relate todeactivating certain functionality for purposes of reducing powerconsumption. It should be understood, however, that actions cancorrespond to other functions that can adjust similar and otherpotential issues with use of the device. For example, certain functions,such as requesting Web page content, searching for content on a harddrive and opening various applications, can take a certain amount oftime to complete. For devices with limited resources, or that have heavyusage, a number of such operations occurring at the same time can causethe device to slow down or even lock up, which can lead toinefficiencies, degrade the user experience and potentially use morepower.

In order to address at least some of these and other such issues,approaches in accordance with various embodiments can also utilizeinformation such as user gaze direction to activate resources that arelikely to be used in order to spread out the need for processingcapacity, memory space and other such resources.

In some embodiments, the device can have sufficient processingcapability, and the imaging element and associated analyticalalgorithm(s) may be sensitive enough to distinguish between the motionof the device, motion of a user's head, motion of the user's eyes andother such motions, based on the captured images alone. In otherembodiments, such as where it may be desirable for the process toutilize a fairly simple imaging element and analysis approach, it can bedesirable to include at least one orientation determining element thatis able to determine a current orientation of the device. In oneexample, the at least one orientation determining element is at leastone single- or multi-axis accelerometer that is able to detect factorssuch as three-dimensional position of the device and the magnitude anddirection of movement of the device, as well as vibration, shock, etc.Methods for using elements such as accelerometers to determineorientation or movement of a device are also known in the art and willnot be discussed herein in detail. Other elements for detectingorientation and/or movement can be used as well within the scope ofvarious embodiments for use as the orientation determining element. Whenthe input from an accelerometer or similar element is used along withthe input from the camera, the relative movement can be more accuratelyinterpreted, allowing for a more precise input and/or a less compleximage analysis algorithm.

When using an imaging element of the computing device to detect motionof the device and/or user, for example, the computing device can use thebackground in the images to determine movement. For example, if a userholds the device at a fixed orientation (e.g. distance, angle, etc.) tothe user and the user changes orientation to the surroundingenvironment, analyzing an image of the user alone will not result indetecting a change in an orientation of the device. Rather, in someembodiments, the computing device can still detect movement of thedevice by recognizing the changes in the background imagery behind theuser. So, for example, if an object (e.g., a window, picture, tree,bush, building, car, etc.) moves to the left or right in the image, thedevice can determine that the device has changed orientation, eventhough the orientation of the device with respect to the user has notchanged. In other embodiments, the device may detect that the user hasmoved with respect to the device and adjust accordingly. For example, ifthe user tilts their head to the left or right with respect to thedevice, the content rendered on the display element may likewise tilt tokeep the content in orientation with the user.

Various approaches can be utilized for locating one or more desiredfeatures of a user's face to determine various aspects useful fordetermining relative orientation. For example, an image can be analyzedto determine the approximate location and size of a user's head or face.FIG. 14(a) illustrates an example wherein the approximate position andarea of a user's head or face 1400 is determined and a virtual “box”1402 is placed around the face as an indication of position using one ofa plurality of image analysis algorithms for making such adetermination. Using one algorithm, a virtual “box” is placed around auser's face and the position and/or size of this box is continuallyupdated and monitored in order to monitor relative user position.Similar algorithms can also be used to determine an approximate locationand area 1404 of each of the user's eyes (or in some cases the eyes intandem). By determining the location of the user's eyes as well,advantages can be obtained as it can be more likely that the imagedetermined to be the user's head actually includes the user's head, andit can be determined that the user is facing the device. Further, therelative movement of the user's eyes can be easier to detect than theoverall movement of the user's head when performing motions such asnodding or shaking the head back and forth. Monitoring box size alsohelps to provide distance information as well as directionalinformation, which can be helpful when generating a three-dimensionalmodel for modifying image information based on relative user position.

Various other algorithms can be used to determine the location offeatures on a user's face. For example, FIG. 14(b) illustrates anexample wherein various features on a user's face are identified andassigned a point location 1406 in the image. The system thus can detectvarious aspects of user features and can determine more subtle changesin orientation. Such an approach provides advantages over the generalapproach of FIG. 14(a) in certain situations, as various other featurescan be determined, in case the user's eyes cannot be seen due toglasses, hair, etc.

Once the positions of facial features of a user are identified, relativemotion between the user and the device can be detected and utilized asinput. For example, FIG. 14(c) illustrates an example where the user'shead 1400 is moving up and down with respect to the viewable area of theimaging element. As discussed, this could be the result of the usermoving his or her head, or the user moving the device up and down, etc.FIG. 14(d) illustrates a similar example wherein the user is movingright to left relative to the device, through movement of the user, thedevice, or both. As can be seen, each movement can be tracked as avertical or horizontal movement, respectively, and each can be treateddifferently as an input to modify a displayed image. As should beunderstood, such a process also can detect diagonal or other suchmovements. FIG. 14(e) further illustrates an example wherein the usertilts the device and/or the user's head, and the relative change in eyeposition is detected as a rotation. In some systems, a “line” thatcorresponds to the relative position of the eyes can be monitored, and ashift in angle of this line can be compared to an angle threshold todetermine when the rotation should be interpreted as input. FIG. 14(f)illustrates another advantage of using an approach such as thatdescribed with respect to FIG. 14(b) to determine the position ofvarious features on a user's face. In this exaggerated example, it canbe seen that the features of a second user's head 1408 have a differentrelative position and separation. Thus, the device also can not onlydetermine positions of features for a user, but can distinguish betweendifferent users.

FIGS. 15(a) and 15(b) illustrate an example approach that can be used todetermine variations in relative distance between a user and a devicethat can be used in accordance with various embodiments. As in FIG.15(a), the approximate position and area of a user's head or face 1500is determined and a virtual “box” 1502 is placed around the face at aninitial distance as an indication of distance using one of a pluralityof image analysis algorithms for making such a determination. If theuser is known, the size of the user's head might be stored such that anactual distance to the user can be calculated based at least in partupon the size of the box 1502. If the user is not known, the distancecan be estimated or determined using other factors, such as stereoscopicimaging. In some embodiments, determinations will be relative withrespect to an initial box size when the actual distance cannot bedetermined

As the distance between the user and the device changes, the size of thevirtual box will change as well. For example, in FIG. 15(b) the distancebetween the user and the device has increased, such that the user's head1520 appears smaller in the captured image information. Accordingly, thesize of the virtual box 1522 for the adjusted size of the user's head issmaller than the original box 1502 for the initial distance. Bymonitoring adjustments in the size of the box or another measure of theuser's head and/or other such features (e.g., boxes 1524), the devicecan determine an approximate distance and/or change in distance to theuser. As discussed, this information can be used to adjust aspects ofthe displayed image information such as a level of zoom or amount ofdetail.

FIGS. 16(a) to 16(d) illustrate an example of how an interface plane orelement at different depths can be used to generate viewing-angleappropriate images in accordance with at least some embodiments. In FIG.16(a), the example orientation 1600 has a user 1602 substantially infront of a display element 1604 of a device. For simplicity ofexplanation, the interface plane or element here is represented in threedimensions, with a box 1606 on a background 1608. At the current viewingangle, the user is only able to see the top surface 1610 of theinterface plane or element 1606, as illustrated in the display view 1620of FIG. 16(b). In the orientation 1640 of FIG. 16(c), the device hasbeen rotated (or the user has moved with respect to the device). Toprovide an appropriate user experience in at least some embodiments, theinterface plane or element is effectively rotated with the device, suchthat the interface plane or element and background 1608 would rotateaccordingly. Based on the current viewing direction of the user 1602, itcan be seen in the display view 1660 of FIG. 16(d) that the viewableportion 1642 of the interface plane or element includes not only the topof the interface plane or element but at a level of depth (i.e., theinterface plane appears to be closer to a display screen of the device).By calculating this angle, the application can determine the portions ofthe top and side of the interface plane or element to display as aresult of the rotation. It also can be seen in FIG. 16(c) that any areaoccluded by the right side of the interface plane or element in FIG.16(a) now can be seen, and that the area occluded by the left side ofthe box is interface plane or element greater in FIG. 16(c).

In at least some embodiments, a computing device can utilize one or morecameras or other such sensors to determine the relative direction of theuser. For example, FIG. 17(a) illustrates an example situation 1700wherein a computing device 1702 is configured to utilize at least onecamera element 1706 to attempt to locate a feature of a user, such asthe user's head or eyes, for purposes of point of view determination. Inthis example, the user's eyes 1704 are located within the field of view1708 of a camera of the computing device 1702. As discussed elsewhereherein, however, the point of view of a user can be determined usingpositions of the user's eyes, pupils, head, or other such features thatcan be indicative of at least a general point of view. In someembodiments, the device might look for an object held by or otherwiseassociated with a user to determine a general point of view forrendering. Further, in some embodiments a device might utilize at leasttwo different cameras positioned on the device with a sufficientseparation such that the device can utilize stereoscopic imaging (oranther such approach) to determine a relative position of one or morefeatures, with respect to the device, in three dimensions. It should beunderstood that there can be additional imaging elements of the same ora different type at various other locations on the device as well withinthe scope of the various embodiments.

Software executing on the computing device (or otherwise incommunication with the computing device) can obtain information such asthe angular field of view of the camera, the zoom level at which theinformation is currently being captured, and any other such relevantinformation, which can enable the software to determine an approximatedirection 1710 of at least one of the user's eyes with respect to thecamera. In many embodiments, direction information will be sufficient toprovide adequate point-of-view dependent rendering. In at least someembodiments, however, it can also be desirable to determine distance tothe user in order to provide a more consistent and accurate rendering.In some embodiments, methods such as ultrasonic detection, feature sizeanalysis, luminance analysis through active illumination, or other suchdistance measurement approaches can be used to assist with positiondetermination. In other embodiments, a second camera can be used toenable distance determinations through stereoscopic imaging. Once thedirection vectors from at least two image capture elements aredetermined for a given feature, the intersection point of those vectorscan be determined, which corresponds to the approximate relativeposition in three dimensions of the respective feature as known fordisparity mapping and other such processes.

Further illustrating such an example approach, FIG. 17(b) illustrates anexample image 1720 that could be captured of the user's head and eyesusing the camera 1706 of FIG. 17(a). One or more image analysisalgorithms can be used to analyze the image to perform patternrecognition, shape recognition, or another such process to identify afeature of interest, such as the user's eyes. Approaches to identifyinga feature in an image, such may include feature detection, facialfeature extraction, feature recognition, stereo vision sensing,character recognition, attribute estimation, or radial basis function(RBF) analysis approaches, are well known in the art and will not bediscussed herein in detail. As illustrated in this example, both eyes ofthe user might be able to be located in the captured image information.At least some algorithms are able to determine an approximate locationor region 1722, 1724 for each eye, or at least an approximate location1728 of the user's head, where at least one of those locations orregions is used for point of view determinations. Depending on factorssuch as the desired level of sensitivity and distance between the userand the device, however, such information can impact the accuracy of thepoint of view determinations. Approaches in accordance with variousembodiments can take advantage of the fact that the human brain combinesand processes information from both eyes to provide a “single” point ofview. Thus, the software can attempt to determine an intermediate point1726 between the user's eyes to use for the user's point of view.Various other approaches can be used as well, such as are discussedelsewhere herein. Once a relative location is determined in the imageinformation, the device can use information such as the field of view ofthe camera, the position of the camera with respect to the device, thezoom level of the camera, and other such information to determine arelative direction of the user, with that relative direction being usedfor the point of view to use in rendering the interface.

When using a camera to track location, however, the accuracy is limitedat least in part by the frame rate of the camera. Further, images takesome time to process such that there can be some lag in thedeterminations. As changes in orientation of the device can occurrelatively quickly, it can be desirable in at least some embodiments toenhance the accuracy of the point of view determinations. In someembodiments, a sensor or other such element of a computing device can beused to determine motions of the computing device, which can help adjustpoint of view determinations. The sensors can be any appropriate sensorscapable of providing information about rotations and/or translations ofthe device, as may include accelerometers, inertial sensors, electronicgyroscopes, electronic compasses, and the like.

For example, FIG. 18(a) illustrates a “top view” 1800 of a computingdevice 1802 operable to capture an image of an object 1804 (e.g., auser's head) within an angular view 1808 of a camera 1810 of thecomputing device. In this example, the computing device 1802 includes atleast one orientation- or rotation-determining element, such as anelectronic compass or electronic gyroscope, that is able to determine aframe of reference 1806 in two or three dimensions with respect to afirst orientation of the device. In at least some embodiments, anelectronic compass might be used to determine an axis of the frame ofreference 1806, as may correspond to a North direction, etc. In otherembodiments, a component such as an electronic gyroscope might becalibrated periodically with a component such as a compass, but mightinstead determine changes in orientation along three axes of rotationover time. Various other approaches to determining changes inorientation along one, two, or three axes of rotation can be used aswell within the scope of the various embodiments.

A first frame of reference 1806 or orientation can be determined at ornear the time of capture of a first image by a camera 1810 of thecomputing device 1802. In some embodiments, the determination can betriggered by receiving input to capture an image or another such action,but in other embodiments the frame of reference and/or orientationinformation can be updated periodically, such as several times a secondbased upon the type and/or configuration of the electronic gyroscope.The gyroscope can also be any appropriate electronic gyroscopecomponent, such as a conventional MEMS gyroscope used in variousconsumer devices. Approaches for implementing and obtaining orientationchanges from such a gyroscope are well known in the art and, as such,will not be discussed in detail herein.

FIG. 18(b) illustrates a second top view 1810 after a change inorientation of the computing device 1802. The electronic gyroscope (orother such component or embedded sensor) can detect the change inorientation, in this example corresponding to a change in angle 1812with respect to the frame of reference in the plane of the figure. Thegyroscope can present information about the change in orientation in anyappropriate form, such as in angles or radians of change for one, two,or three degrees (e.g., Δx, Δy, Δz), percentage changes in pitch, roll,and yaw, etc. In this example, the change in orientation is determinedto be a given angular amount of rotation 1812 about a single axis. Asillustrated, this causes the object 1804 to be moved to the right edgeof the field of view 1808 of the camera 1810. In at least someembodiments, the gyroscope may not be accurate enough to provide anexact amount of rotation, but can provide an approximation or estimateof the amount of rotation that can be used to narrow the search spaceand facilitate the location of corresponding objects in the images.Further, the information can provide a faster adjustment or predictionof relative position than can be provided from the camera in at leastsome embodiments. A similar approach can be used for translation,although the effects of translation on objects in captured images can bemuch less significant than angular changes, such that the imageinformation might be sufficient to account for translation changes in atleast some embodiments.

As discussed, different approaches can be implemented in variousenvironments in accordance with the described embodiments. For example,FIG. 19 illustrates an example of an environment 1900 for implementingaspects in accordance with various embodiments. As will be appreciated,although a Web-based environment is used for purposes of explanation,different environments may be used, as appropriate, to implement variousembodiments. The system includes electronic client devices 1918, 1920,1922, and 1924, which can include any appropriate device operable tosend and receive requests, messages or information over an appropriatenetwork 1904 and convey information back to a user of the device.Examples of such client devices include personal computers, cell phones,handheld messaging devices, laptop computers, set-top boxes, personaldata assistants, electronic book readers and the like. The network caninclude any appropriate network, including an intranet, the Internet, acellular network, a local area network or any other such network orcombination thereof. The network could be a “push” network, a “pull”network, or a combination thereof. In a “push” network, one or more ofthe servers push out data to the client device. In a “pull” network, oneor more of the servers send data to the client device upon request forthe data by the client device. Components used for such a system candepend at least in part upon the type of network and/or environmentselected. Protocols and components for communicating via such a networkare well known and will not be discussed herein in detail. Communicationover the network can be enabled via wired or wireless connections andcombinations thereof. In this example, the network includes theInternet, as the environment includes a Web server 1906 for receivingrequests and serving content in response thereto, although for othernetworks, an alternative device serving a similar purpose could be used,as would be apparent to one of ordinary skill in the art.

The illustrative environment includes at least one application server1908 and a data store 1910. It should be understood that there can beseveral application servers, layers or other elements, processes orcomponents, which may be chained or otherwise configured, which caninteract to perform tasks such as obtaining data from an appropriatedata store. As used herein, the term “data store” refers to any deviceor combination of devices capable of storing, accessing and retrievingdata, which may include any combination and number of data servers,databases, data storage devices and data storage media, in any standard,distributed or clustered environment. The application server 1908 caninclude any appropriate hardware and software for integrating with thedata store 1910 as needed to execute aspects of one or more applicationsfor the client device and handling a majority of the data access andbusiness logic for an application. The application server providesaccess control services in cooperation with the data store and is ableto generate content such as text, graphics, audio and/or video to betransferred to the user, which may be served to the user by the Webserver 1906 in the form of HTML, XML or another appropriate structuredlanguage in this example. The handling of all requests and responses, aswell as the delivery of content between the client devices 1918, 1920,1922, and 1924 and the application server 1908, can be handled by theWeb server 1906. It should be understood that the Web and applicationservers are not required and are merely example components, asstructured code discussed herein can be executed on any appropriatedevice or host machine as discussed elsewhere herein.

The data store 1910 can include several separate data tables, databasesor other data storage mechanisms and media for storing data relating toa particular aspect. For example, the data store illustrated includesmechanisms for storing content (e.g., production data) 1912 and userinformation 1916, which can be used to serve content for the productionside. The data store is also shown to include a mechanism for storinglog or session data 1914. It should be understood that there can be manyother aspects that may need to be stored in the data store, such as pageimage information and access rights information, which can be stored inany of the above listed mechanisms as appropriate or in additionalmechanisms in the data store 1919. The data store 1919 is operable,through logic associated therewith, to receive instructions from theapplication server 1908 and obtain, update or otherwise process data inresponse thereto. In one example, a user might submit a search requestfor a certain type of item. In this case, the data store might accessthe user information to verify the identity of the user and can accessthe catalog detail information to obtain information about items of thattype. The information can then be returned to the user, such as in aresults listing on a Web page that the user is able to view via abrowser on anyone of the user devices 1918, 1920, 1922 and 1924.Information for a particular item of interest can be viewed in adedicated page or window of the browser.

Each server typically will include an operating system that providesexecutable program instructions for the general administration andoperation of that server and typically will include computer-readablemedium storing instructions that, when executed by a processor of theserver, allow the server to perform its intended functions. Suitableimplementations for the operating system and general functionality ofthe servers are known or commercially available and are readilyimplemented by persons having ordinary skill in the art, particularly inlight of the disclosure herein.

The environment in one embodiment is a distributed computing environmentutilizing several computer systems and components that areinterconnected via communication links, using one or more computernetworks or direct connections. However, it will be appreciated by thoseof ordinary skill in the art that such a system could operate equallywell in a system having fewer or a greater number of components than areillustrated in FIG. 19. Thus, the depiction of the system 1900 in FIG.19 should be taken as being illustrative in nature and not limiting tothe scope of the disclosure.

The various embodiments can be further implemented in a wide variety ofoperating environments, which in some cases can include one or more usercomputers or computing devices which can be used to operate any of anumber of applications. User or client devices can include any of anumber of general purpose personal computers, such as desktop or laptopcomputers running a standard operating system, as well as cellular,wireless and handheld devices running mobile software and capable ofsupporting a number of networking and messaging protocols. Such a systemcan also include a number of workstations running any of a variety ofcommercially-available operating systems and other known applicationsfor purposes such as development and database management. These devicescan also include other electronic devices, such as dummy terminals,thin-clients, gaming systems and other devices capable of communicatingvia a network.

Most embodiments utilize at least one network that would be familiar tothose skilled in the art for supporting communications using any of avariety of commercially-available protocols, such as TCP/IP, OSI, FTP,UPnP, NFS, CIFS and AppleTalk. The network can be, for example, a localarea network, a wide-area network, a virtual private network, theInternet, an intranet, an extranet, a public switched telephone network,an infrared network, a wireless network and any combination thereof.

In embodiments utilizing a Web server, the Web server can run any of avariety of server or mid-tier applications, including HTTP servers, FTPservers, CGI servers, data servers, Java servers and businessapplication servers. The server(s) may also be capable of executingprograms or scripts in response requests from user devices, such as byexecuting one or more Web applications that may be implemented as one ormore scripts or programs written in any programming language, such asJava®, C, C# or C++ or any scripting language, such as Perl, Python orTCL, as well as combinations thereof. The server(s) may also includedatabase servers, including without limitation those commerciallyavailable from Oracle®, Microsoft®, Sybase® and IBM®.

The environment can include a variety of data stores and other memoryand storage media as discussed above. These can reside in a variety oflocations, such as on a storage medium local to (and/or resident in) oneor more of the computers or remote from any or all of the computersacross the network. In a particular set of embodiments, the informationmay reside in a storage-area network (SAN) familiar to those skilled inthe art. Similarly, any necessary files for performing the functionsattributed to the computers, servers or other network devices may bestored locally and/or remotely, as appropriate. Where a system includescomputerized devices, each such device can include hardware elementsthat may be electrically coupled via a bus, the elements including, forexample, at least one central processing unit (CPU), at least one inputdevice (e.g., a mouse, keyboard, controller, touch-sensitive displayelement or keypad) and at least one output device (e.g., a displaydevice, printer or speaker). Such a system may also include one or morestorage devices, such as disk drives, optical storage devices andsolid-state storage devices such as random access memory (RAM) orread-only memory (ROM), as well as removable media devices, memorycards, flash cards, etc.

Such devices can also include a computer-readable storage media reader,a communications device (e.g., a modem, a network card (wireless orwired), an infrared communication device) and working memory asdescribed above. The computer-readable storage media reader can beconnected with, or configured to receive, a computer-readable storagemedium representing remote, local, fixed and/or removable storagedevices as well as storage media for temporarily and/or more permanentlycontaining, storing, transmitting and retrieving computer-readableinformation. The system and various devices also typically will includea number of software applications, modules, services or other elementslocated within at least one working memory device, including anoperating system and application programs such as a client applicationor Web browser. It should be appreciated that alternate embodiments mayhave numerous variations from that described above. For example,customized hardware might also be used and/or particular elements mightbe implemented in hardware, software (including portable software, suchas applets) or both. Further, connection to other computing devices suchas network input/output devices may be employed.

Storage media and computer readable media for containing code, orportions of code, can include any appropriate media known or used in theart, including storage media and communication media, such as but notlimited to volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information such as computer readable instructions, data structures,program modules or other data, including RAM, ROM, EEPROM, flash memoryor other memory technology, CD-ROM, digital versatile disk (DVD) orother optical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices or any other medium which canbe used to store the desired information and which can be accessed by asystem device. Based on the disclosure and teachings provided herein, aperson of ordinary skill in the art will appreciate other ways and/ormethods to implement the various embodiments.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims.

What is claimed is:
 1. A computing device, comprising: a display screen;a camera; at least one computing device processor; a memory deviceincluding instructions that, when executed by the at least one computingdevice processor, enable the computing device to: determine relationshipinformation between related content capable of being displayed on adisplay screen of a computing device, the relationship informationindicates a relative arrangement of the related content; determine,based at least in part on the relationship information, a node hierarchyfor a plurality of nodes that includes position information for a firstnode of the plurality of nodes, the position information includeslateral position information and depth position information; associate afirst plane of content of a plurality of planes of content with thefirst node of the plurality of nodes based at least on the relationshipinformation and a depth position of the first node; display at a firstlateral position on the display screen, first content included in thefirst plane of content; display at a second lateral position on thedisplay screen, second content included in a second plane of content;acquire, using the camera, at least one image of a viewer of thecomputing device; determine, by analyzing the at least one image, alocation of the viewer with respect to the computing device; determine afirst lateral offset position for the first plane of content based atleast in part on the location of the viewer; determine a second lateraloffset position for the second plane of content based at least in parton the location of the viewer; display, on the display screen, the firstcontent at the first lateral offset position; and display, on thedisplay screen, the second content at the second lateral offsetposition.
 2. The computing device of claim 1, wherein the relatedcontent includes at least first related content and second relatedcontent, wherein the instructions, when executed, further cause thecomputing device to: determine a change in position of the first relatedcontent caused by at least one of an animation of the first content or ascrolling of an interface displaying the first content; update firstposition information for one or more nodes associated with one of theplurality of planes of content associated with the first content; andupdate second position information for at least one other node of theplurality of nodes associated with at least one other plane of contentbased at least upon the change in position of the first content, the atleast one other plane of content associated with the second content,wherein the at least one other plane of content is determined based atleast in part upon the relationship information.
 3. The computing deviceof claim 1, wherein each plane of the plurality of planes of contentcorresponds to a node of the plurality of nodes of a hierarchy, whereinthe plurality of nodes of the hierarchy include at least one parent nodeand one or more child nodes depending from the at least one parent node,the depth position of the one or more child nodes being relative to atleast one of the display screen or the at least one parent node.
 4. Thecomputing device of claim 1, wherein the instructions, when executed,further cause the computing device to: render at least one shadow basedat least in part on an intersection of light from a virtual light sourcewith the first plane of content.
 5. A computer implemented method,comprising: determining relationship information between first contentand second content capable of being displayed on a display screen of acomputing device, the first content associated with a first interfaceplane of a plurality of interface planes and the second contentassociated with a second interface plane of the plurality of interfaceplanes; determining, based at least in part on the relationshipinformation, a node hierarchy for a plurality of nodes that includes afirst node associated with the first interface plane and a second nodeassociated with the second interface plane, the first node having firstposition information that includes lateral position information anddepth position information; determining a first lateral offset positionfor the first interface plane based at least on a position of thecomputing device relative to a viewer of the display screen; determininga second lateral offset position for the second interface plane based atleast in part on the position of the computing device relative to theviewer of the display screen; displaying, on the display screen, thefirst content based at least in part on the depth position informationof the first node and the first lateral offset position; and displaying,on the display screen, the second content based at least in part on thesecond lateral offset position.
 6. The computer implement method ofclaim 5, further comprising: determining a change in position of thefirst content caused by at least one of an animation of the firstcontent or a scrolling of an interface displaying the first content;updating the first position information for the first node based atleast on the change in position; and updating second positioninformation for the second node associated with the second interfaceplane based at least on the change in position of the first content. 7.The computer implement method of claim 6, further comprising:determining the second node associated with the second interface planebased at least in part on relationship information between the firstcontent and the second content.
 8. The computer implemented method ofclaim 5, wherein displaying the first content further includes:displaying the first content to appear to be closer to a viewer of thecomputing device than content associated with at least one otherinterface plane, wherein the at least one other interface plane isassociated with at least one other node having a depth positiondifferent than the depth position of the first node.
 9. The computerimplemented method of claim 5, wherein displaying the first contentfurther includes: determining an adjusted appearance of at least one ofa size, shape, color, shading, or blur for at least one of the pluralityof interface planes according to a location of the viewer.
 10. Thecomputer implemented method of claim 5, further comprising: determininga change in position of the computing device from a first location to asecond location relative to the viewer of the display screen; anddetermining the first lateral offset position for the first interfaceplane based at least on the change in position of the computing devicefrom the first location to the second location relative to the viewer todisplay occluded portions of the second content.
 11. The computerimplemented method of claim 5, further comprising: assigning the firstinterface plane to the depth position of the first node of the pluralityof nodes; assigning the second interface plane to a second depthposition associated with a second node of the plurality of nodes, thedepth position of the first node different from the second depthposition of the second node; determining a change in the position of thecomputing device relative to the viewer of the display screen;determining the first lateral offset position for the first interfaceplane further based on the change in the position of the computingdevice relative to the viewer of the display screen; determining thesecond lateral offset position for the second interface plane furtherbased on the change in the position of the computing device relative tothe viewer of the display screen; and displaying, on the display screen,the second content further based at least in part on the second depthposition associated with the second node.
 12. The computer implementedmethod of claim 5, rendering, based at least on a draw order, the firstinterface plane of the plurality of interface planes with a first depthposition before rendering the second interface plane of the plurality ofinterface planes with a second depth position, the first interface planeappearing closer to a viewer of the computing device than the secondinterface plane.
 13. The computer implemented method of claim 5, furthercomprising: determining, based at least in part on the location of theviewer, coefficients for a set of transformation equations to adjust ascale and a translation for a respective interface plane with respect toa determined pivot point associated with a center of the display screen;and determining an adjusted appearance of for the first contentassociated with the first interface plane using the set oftransformation equations.
 14. The computer implemented method of claim5, wherein each interface plane of the plurality of interface planescorresponds to a node of the plurality of nodes of the node hierarchy,wherein the plurality of nodes include at least one parent node and oneor more child nodes depending from the at least one parent node, andwherein the plurality of nodes are provided via an operating system ofthe computing device.
 15. The computer implemented method of claim 5,further including: rendering at least one shadow based at least in parton an intersection of light from a virtual light source with the firstinterface plane.
 16. A non-transitory computer readable storage mediumstoring one or more sequences of instructions executable by one or moreprocessors to perform a set of operations comprising: determiningrelationship information between first content and second contentcapable of being displayed on a display screen of a computing device,the first content being associated with a first interface plane of aplurality of interface planes and the second content being associatedwith a second interface plane of the plurality of interface planes;determining, based at least in part on the relationship information, anode hierarchy for a plurality of nodes that includes a first nodeassociated with the first interface plane and a second node associatedwith the second interface plane, the first node of the plurality ofnodes having position information that includes lateral positioninformation and depth position information; determining a first lateraloffset position for the first interface plane based at least on aposition of the computing device relative to a viewer of the displayscreen; determining a second lateral offset position for the secondinterface plane based at least in part on the position of the computingdevice relative to the viewer of the display screen; and displaying, onthe display screen, the first content based at least in part on thedepth position information of the first node and the first lateraloffset position for the first interface plane; and displaying, on thedisplay screen, the second content based at least in part on the secondlateral offset position for the second interface plane.
 17. Thenon-transitory computer readable storage medium of claim 16, furthercomprising instructions executed by the one or more processors toperform the operations of: determining a change in position of the firstcontent caused by at least one of an animation of the first content or ascrolling of an interface displaying the first content; updating thefirst lateral offset position for the first interface plane based atleast on the change in position; and updating the second lateral offsetposition for the second interface plane based at least on the change inposition.
 18. The non-transitory computer readable storage medium ofclaim 16, further comprising instructions executed by the one or moreprocessors to perform the operations of: determining a change inposition of the computing device from a first location to a secondlocation relative to the viewer of the display screen; and determiningthe first lateral offset position for the first interface plane based atleast on change in position of the computing device from the firstlocation to the second location relative to the viewer to displayoccluded potions of the second content.
 19. The non-transitory computerreadable storage medium of claim 16, wherein a first element associatedwith the first interface plane of the plurality of interface planes iscapable of partially obscuring a second element associated with thesecond interface plane of the plurality of interface planes.